spec: Add AudioWorklet engine spec for scheduling, LFO, metering, and pitch-shifting#45
Open
spec: Add AudioWorklet engine spec for scheduling, LFO, metering, and pitch-shifting#45
Conversation
… pitch-shifting Defines four AudioWorklet modules to move performance-critical audio work off the main thread, plus a metrics infrastructure to measure impact: 1. Scheduler Worklet — moves timing loop to audio thread (jitter <1ms) 2. Shared LFO Worklet — single LFO for all voices (~25% CPU reduction) 3. Track Metering Worklet — per-track RMS/peak at 60fps 4. Pitch-Shifting Worklet — PSOLA for duration-preserving transposition 5. Metrics infrastructure — jitter, latency, CPU, and quality benchmarks https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
…ration Implements the full AudioWorklet system from AUDIOWORKLET-ENGINE.md: Metrics infrastructure (Phase W1): - RingBuffer: fixed-size ring buffer for metric samples - percentile/mean/stddev: statistical helpers - AudioMetricsCollector: jitter, latency, CPU, drift tracking with configurable sampling rate and PerformanceObserver long tasks - 30 unit tests (all passing) Four AudioWorklet processors: - scheduler.worklet.ts: audio-thread timing loop with drift-free scheduling, swing, tied notes, loop regions, and jitter reporting - shared-lfo.worklet.ts: single LFO for all voices with per-sample (a-rate) waveform computation, replacing 8 per-voice Tone.LFO instances - metering.worklet.ts: per-track RMS/peak/clipping at ~60Hz update rate - pitch-shift.worklet.ts: granular PSOLA for duration-preserving shifts Main-thread hosts: - IScheduler interface for seamless main-thread/worklet swapping - SchedulerWorkletHost: receives note/step/beat events, dispatches to AudioEngine play methods, records jitter metrics - MeteringHost: track connection management, level subscriptions - useTrackMeter/useAllTrackMeters: React hooks for meter UI Engine integration: - AudioEngine.initializeWorklets() loads metering on startup - Metrics providers wired for voice utilization and context health - supportsWorklets(), getAudioMetrics(), getMeteringHost() accessors - Graceful fallback when AudioWorklet not supported Debug API: - audioDebug.metrics(): formatted console output of all metrics - audioDebug.resetMetrics(): clear collected data - audioDebug.workletStatus(): worklet support and health check https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
Bugs found and fixed during line-by-line audit: 1. scheduler.worklet.ts: Jitter metric was comparing currentTime to intended step time — meaningless inside the worklet (timing is sample-accurate by definition). Changed to measure scheduling precision (nextStepTime vs multiplicative reference). Also reduced jitter message rate from every step to every 8th step to avoid flooding the MessagePort. 2. scheduler-worklet-host.ts: Volume reset timers (setTimeout) were not tracked or cleaned up on stop(). Added pendingTimers Set with tracking/cleanup matching the original scheduler.ts:202-206. 3. shared-lfo.worklet.ts: Amplitude tremolo scaling was wrong — amount was pre-multiplied into the waveform before the destination scaling, causing tremolo depth to be too shallow at low amounts. Separated raw waveform from amount and fixed the amplitude formula so amount=1 gives full 0..1 gain swing, amount=0.5 gives 0.5..1.0. 4. metering.worklet.ts: Hardcoded block size of 128 in sample count. The Web Audio spec doesn't guarantee 128-sample blocks. Now reads actual block size from the input channel length. https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
6 new property test files with 68 tests covering: - RingBuffer: capacity bounds, insertion order, overflow FIFO, clear/push - Percentile/mean/stddev: monotonicity, bounds, translation invariance - AudioMetricsCollector: sampling rates, snapshot percentile ordering, reset, drift lookup - LFO waveforms: range bounds, symmetry, destination scaling, amount=0 identity - Scheduler parity: worklet step duration/swing/tied duration match canonical - Pitch-shift: Hann window symmetry/endpoints/overlap-add, pitch ratio invertibility https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
Findings from audio engine review uncovered several performance and configuration issues hiding in plain sight: Config cleanup: - Remove dead test block from vite.config.ts (caused phantom failures when vitest was run from repo root instead of app/) - Remove orphaned vitest.integration.config.ts (never referenced in package.json; real config lives in test/integration/) Performance fixes: - Waveform.tsx: Cache peak computation in WeakMap keyed by (buffer, width) to avoid re-scanning entire Float32Array on every render - CursorOverlay.tsx: Replace setInterval(500ms) tick with CSS transition class, eliminating unnecessary React re-renders for fade animation - canonicalHash.ts: Memoize hashState() to skip redundant JSON.stringify when called repeatedly with unchanged state during multiplayer sync - patternOps.ts: Replace JSON.stringify group comparison in Bjorklund's algorithm with direct array equality check New tooling: - detect-main-thread-hotspots.ts script scans src/ for timers in components, JSON.stringify in hooks, typed array loops in render paths, disabled feature flags, and config conflicts Lessons Learned doc updated with Lesson 19 (config discrepancies) and Lesson 20 (off-main-thread opportunities hide in plain sight). All 4373 tests pass (24 new tests added). https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
MIDI export Web Worker: - midiExport.worker.ts runs exportToMidi() off the main thread - downloadMidi() now dispatches to the worker via postMessage - Falls back to synchronous main-thread export if Worker fails - 8 new tests verify worker protocol, fallback, and MIDI output Dead code audit: - detect-unused-exports.ts script finds 253 unused exports across 76 files (many are type-only; key runtime findings below) - dead-code-audit.test.ts codifies known dead code as living docs: Built but never wired in: - SchedulerWorkletHost: full implementation, never instantiated - isWorkletSchedulerEnabled(): exported, never imported - synthesis.ts: 22 pure synthesis functions, only used by tests - note-player.ts notePlayerRegistry: only used by tests - useTrackMeter hook: never used in any component These are tracked as test assertions so they break loudly if someone accidentally re-introduces or removes them without updating the audit. All 4392 tests pass (19 new tests added). https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
toBeCloseTo(val, 10) fails on large doubles due to IEEE 754 rounding (e.g., -262270.89... had 5.82e-11 error vs 5e-11 threshold). Reduce to 8 decimal digits — still extremely precise, but immune to FP edge cases. https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
b0fc9f7 to
e724fb1
Compare
synthesis.ts duplicated all 22 instrument synthesis functions already implemented in samples.ts via Web Audio API. Zero non-test imports confirmed — only synthesis.test.ts referenced it. https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
Strategy/Chain pattern for note dispatch that was never wired in. The scheduler's switch(instrumentType) with direct audioEngine calls has been the production path since day one. https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
The function read VITE_WORKLET_SCHEDULER but was never called. The worklet scheduler feature flag will be managed through the centralized features.ts config instead. https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
Flip features.loopRuler default from false to true. The LoopRuler UI component was already imported and conditionally rendered in StepSequencer, and the loopRegion state is integrated across 21 files (scheduler, state, multiplayer, TrackRow dimming). https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
Connect track audio buses to the metering worklet so useTrackMeter() returns real data. Add TrackMeter component rendering vertical VU bars with RMS level, peak hold, and clipping indicator. Changes: - TrackBus: add getOutputNode() accessor for metering tap - TrackBusManager: auto-connect/disconnect tracks to metering - TrackMeter: new component consuming useTrackMeter hook - MixerPanel: render TrackMeter next to each channel fader https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
…e-render storm - Fix #3: Replace shared sampleCount with per-track Uint32Array in metering worklet so tracks with partial audio get correct RMS (not artificially low) - Fix #2: Add freeIndices pool to MeteringHost so disconnected track indices are reclaimed instead of leaking (was exhausting after 16 cycles) - Fix #1/#9: Add rAF coalescing + value-change threshold to useTrackMeter to reduce re-renders from ~60/sec to at most display refresh rate - Fix #7: Remove unused audioContext field from MeteringHost - Fix #10: disconnectTrack now accepts optional busOutput param to explicitly disconnect the audio graph, not just clear internal maps https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
…lication - Fix #4: Add `if (!this.isRunning) return` guard in volume reset timer callbacks in both Scheduler and SchedulerWorkletHost to prevent stale resets firing after stop() - Fix #5: Add cross-reference comments between scheduler-types.ts and scheduler.worklet.ts documenting intentional type duplication (worklets can't import external modules) https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
- Fix #6: Extract MidiExportOptions, MidiWorkerRequest, MidiWorkerResponse, MidiWorkerError to midiExport.types.ts, breaking the circular dependency between midiExport.ts and midiExport.worker.ts - Fix #8: Add early return in computePeaks when width <= 0 to prevent infinite loop (Math.ceil(length / 0) = Infinity) https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
- Add workletScheduler feature flag (VITE_FEATURE_WORKLET_SCHEDULER, default: false) - Make Scheduler implement IScheduler interface for polymorphism - Add upgradeToWorkletScheduler() that dynamically imports SchedulerWorkletHost and replaces the singleton when flag is on and browser supports AudioWorklet - Call upgrade from AudioEngine.initializeWorklets() after metering init - Update dead-code audit: SchedulerWorkletHost is now wired (dynamic import) - ES module live binding ensures all consumers see the upgraded scheduler https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
- Import XYPadController from audio/xyPad.ts into Transport.tsx - Add preset selector dropdown with 6 presets (filter-sweep, lfo-control, envelope-shape, space-control, delay-modulation, oscillator-filter) - Map effect-related XYPadParameters (reverbWet, delayWet, delayFeedback, chorusWet, distortionWet) to updateEffect calls - Add XY pad UI in effects panel as a 5th grid item spanning 2 columns - Add CSS for preset selector and XY controller section - Update dead-code audit: XYPadController is now used by Transport https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
- Load pitch-shift worklet in initializeWorklets() alongside metering - Route samples through pitch-shift AudioWorkletNode when shift > ±6 semitones for better quality (grain-based processing vs playbackRate) - Fall back to native playbackRate for smaller shifts (proven quality) - Set pitch via AudioParam 'pitchRatio' (k-rate, range 0.25-4.0) - Clean up worklet node on playback end to prevent memory leaks https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
- Load shared-lfo-worklet in engine.initializeWorklets() - Expose isSharedLfoAvailable() and getAudioContext() on AudioEngine - Create shared LFO AudioWorkletNode in AdvancedSynthEngine.initialize() with 8-channel output (one per voice slot) - Sync worklet config (frequency, waveform, amount, destination) via postMessage when preset changes in setPreset() - Per-voice Tone.LFO remains as primary modulation path; shared worklet runs in parallel for future voice optimization - Clean up worklet node on dispose https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
Remove dependency on lazyAudioLoader in audioTriggers, SamplePicker, audio-debug, and audio-health-canary. Use direct audioEngine import from engine module instead, simplifying the code path and removing the async getAudioEngine() indirection. https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
- Lazy-load StepSequencer with Suspense + StepSequencerSkeleton - Make audio-debug a dev-only dynamic import - Delete lazyAudioLoader.ts (no longer needed — all consumers now import audioEngine directly from engine.ts) - Update dead-code audit test to verify workletScheduler flag https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8
Tone.js and Advanced synths bypassed TrackBusManager, so their tracks never received metering data. Now both engine play methods accept a trackId parameter and dynamically reroute their shared output to the correct track bus before each note. Both schedulers pass trackId for all instrument types. Also fixes the track-meter CSS overflow that caused faders to collapse when the meter expanded beyond its flex container, and adds an onReady() pattern to MeteringHost to eliminate the race condition where useTrackMeter hooks mounted before the worklet was loaded. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Exclude *.worklet.ts from tsconfig.app.json (AudioWorklet globals are unavailable in the main-thread TS context) - Extract shared MIDI worker types to midiExport.types.ts to break circular import between midiExport.ts and midiExport.worker.ts - Add worklet-url.d.ts type declaration for Vite ?worker&url imports - Guard Waveform computePeaks against width <= 0 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Playwright's base.extend() requires the first argument to use object
destructuring ({}), not a regular parameter name. Add eslint-disable
for the unavoidable empty-pattern.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ts, ref-during-render - Remove duplicate getAudioContext() in engine.ts (lines 76 vs 602) - Remove unused fileExists function in dead-code-audit.test.ts - Mock requestAnimationFrame in useTrackMeter race tests (jsdom doesn't flush rAF inside act()) - Move XY preset ref update from render body to event handler in Transport.tsx to satisfy react-hooks/refs lint rule Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update feature flag test: loop ruler is now ON by default - Fix landscape alignment tests: use different instruments for multi-track - Exclude VU meter mutations from playback flicker observer - Update visual regression snapshots after UI changes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…rarchy Replace the bespoke Reverb XY Pad and broken per-parameter switch with a unified system that batches all parameter changes into a single state update, preventing the stale-closure bug where the Nth updateEffect() call overwrote the (N-1)th. Architecture: - xy-effects-bridge.ts: buildBatchedEffectsUpdate() collects all XY param changes into one EffectsState, applySynthParam() routes filter/ LFO/envelope/oscMix to AdvancedSynthEngine - reverb-control preset replaces bespoke handleReverbXY (proven identical via property-based tests with 500 randomized inputs per property) - AdvancedSynthEngine gains 7 individual parameter setters for real-time XY control (filter freq/Q, LFO rate/amount, attack/release, osc mix) Fixes: - 4 of 6 XY presets were silently dropping params (filter-sweep, lfo-control, envelope-shape, oscillator-filter) — now fully wired - Stale closure: space-control drag lost reverbWet when delayWet was applied second — now single batched state update - Redundant Reverb XY Pad removed (was a separate code path doing what the generic system should have done) Layout: - XY Pad moved from bottom to top of FX panel — it's the macro control, the sliders below are the detail controls - CSS nth-child color selectors updated to match new DOM order Tests (76 new across 5 files): - 33 property-based tests: state machine, curve monotonicity, parameter clamping, routing exclusivity, oscMix inverse, idempotence, continuity, preset switching safety - 25 unit tests: batched updates, synth routing, param classification - 7 backward compat tests: reverb-control ≡ old handleReverbXY - 7 bridge property tests: range, batching, roundtrip - 4 component tests: Transport batched drag, no stale closure Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Captures insights on bespoke one-offs as tech debt, silent failures, dead API surface as a signal, the React stale closure anti-pattern, layout matching interaction hierarchy, property-based testing, and proving backward compatibility with PBT. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add default warnings to effects-util.ts and xy-effects-bridge.ts switches so unknown params/effects log instead of silently dropping - Create shared effect-param-mapping.ts to eliminate duplicate routing tables between effects-util and xy-effects-bridge - Add PBT for semitoneToFrequency (monotonicity, positivity, octave doubling) - Add equivalence proof for arraysEqual vs JSON.stringify - Export arraysEqual from patternOps.ts for testability - Remove dead WaveformType re-export from advancedSynth.ts - Fix flaky percentile monotonicity PBT (epsilon 1e-10 → 1e-6) - Fix flaky playable-range touching test (add 'equal' to expected set) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Defines four AudioWorklet modules to move performance-critical audio work
off the main thread, plus a metrics infrastructure to measure impact:
https://claude.ai/code/session_01SMDAri4k9iSxMrkZP7HXu8